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Abstract 

Simonson, Schlosser and Hanson ( 1 999) argue that a new theory called “equivalency theory ” is 
needed to account for the unique features of the “teleconferencing” ( synchronous ) model ofDE that is 
prevalent in many North American universities. Based on a comprehensive meta-analysis of the comparative 
literature ofDE (Bernard, Abrami, Lou, Wozney, Borokhovski, Wallet, Wade, Fiset, & Huang, in press), we are 
able to assess empirically whether equivalency has been achieved in prior comparative DE research. This 
paper includes a brief summary of the results of the split between synchronous and asynchronous patterns of 
DE, and addresses the implications these data have for developing separate theories of DE for synchronous 
(i.e., group-based) and asynchronous (i.e., individualized ) applications. We examine data based on 
achievement, attitude and retention outcomes and coded study features (i.e., methodological, pedagogical and 
media) relating to them. 



Introduction 

Over the past several decades, two distinctly different patterns of distance education (DE) have 
emerged, along with a variety of combinations of them. Synchronous DE derives from early applications of 
closed circuit TV on college campuses (e.g.. Carpenter & Greenhill, 1955; 1958). In this pattern, two or more 
classrooms in different locations are joined in real time and run, synchronously, usually from the originating 
site. Today, various forms of audio and video interactive teleconferencing technology are used to unite 
originating and remote classroom sites. According to Mottet (1998) and Ostendorf (1997), this form of 
“emulated traditional classroom instruction” is the fastest growing form ofDE in U.S. universities, and so it is 
important for us to know how it affects learners who are involved in it. 

Modern asynchronous DE is a derivative of correspondence education, where the “medium of 
instruction” was the post office and asynchronicity was a result of postal delay. In this pattern, students in 
remote locations work independently or in asynchronous groups, usually with the support of an instructor or 
tutor. Communication between the instructor and student and among students typically involves the Internet, 
although contact through other media is (e.g., telephone) is often used. Asynchronous DE, then, fits within 
Keegan’s (1996) definitional criteria of teaching and learning “anyplace, anytime,” whereas synchronous DE is 
both time and place dependent. 
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Simonson, Schlosser and Hanson ( 1999) argue that separate theories of DE are needed to account for 
the different underlying premises of these two models. What we have referred to as asynchronous DE springs 
from theoretical perspectives that emphasize “independence and autonomy of the learner, industrialization of 
teaching, and interaction and communication. These classical theories emphasize the notion that distance 
education is a fundamentally different form of education’’ (p. 74). By contrast, Simonson et al. argue that a 
theory of synchronous DE should focus on providing individual but equivalent experiences for the classroom 
and DE learners. They say that “Recent emerging theories based on the capabilities of new interactive 
telecommunication-based audio and video systems suggest that distance education may not be a distinct field of 
education” (p. 74). 

We can take no position on the theoretical arguments contained in the above quotations, except to say 
that there is one important distinction that is evident in the two patterns. Synchronous DE, by definition, is 
inextricably tied to classroom instruction. In this sense, it may be viewed as an extension of, or as a special case 
of classroom instruction. Asynchronous DE, by contrast, is not necessarily bound by conditions that exist in a 
classroom. This fact suggests that the two patterns may encompass different, but related, sets of teaching and 
learning skills. 

An issue that arises is upon what grounds should “equivalence” be judged. Simonson et al. (1999) offer 
this suggestion: “... the outcomes of a learning experience are those obvious, measurable, and significant 
changes that occur cognitively and affectively in learners because of their participation in the course or unit” (p. 
72). They distinguish between “instructor-determined outcomes” and “learner-determined outcomes.” The 
former are presumably the traditional measures of achievement and course satisfaction (e.g., attitude measures), 
while the latter are delayed measures of skills application and follo\wup enrollment in similar courses. We 
argue that another related measure of learner-determination is the retention rate (or conversely, the dropout rate) 
in a course. 

The research literature of DE contains many studies of both synchronous and asynchronous DE as they 
compare to traditional classroom instruction. In 2001, we undertook to review, quantitatively (meta-analysis), 
these comparative studies dating from 1985 through 2002. These were some of the findings: 1) there is wide 
variability in achievement, attitude and retention outcomes (i.e., some classrooms vastly outperform DE and 
vice versa); 2) study features describing differences in research methodology account for a substantial 
proportion of the variance in outcomes; 3) overall, elements of pedagogy account for more variation than 
elements of media usage (Clark, 1983, 1994); and 4) a number of study features (both pedagogy and media) are 
predictors of the differences between DE and classroom instruction (Bernard, Abrami, Lou, Wozney, 
Borokhovski, Wallet, Wade, Fiset, & Huang, in press). Here we will present a subset of these results relating to 
synchronous and asynchronous DE to describe how these patterns differ in terms of achievement, attitude and 
retention outcomes. We will focus especially on the comparative effects of synchronous DE and consider 
whether the DE part of this dyad differs from its classroom counterpart. We will also examine the effects of 
individual pedagogy and media study features in an attempt to suggest what might be changed in synchronous 
DE, in particular to increase its “equivalence” to classroom instruction. 

The Nature of Meta- Analysis 

An axiom of modern cognitive science is that knowledge is situated. Likewise, the evidence from a 
primary investigation should also be situated in the larger context of other investigations so that evidence is 
organized, accumulated and otherwise synthesized — a cornerstone of any science. Meta-analysis, or quantitative 
research synthesis (Glass, McGaw & Smith, 1981; Hedges & Olkin, 1985), is the most unbiased way of 
conducting a systematic review and estimating the treatment effects associated with a collection of primary 
studies addressing different facets of a common question. The methodology also provides ways of asking and 
answering more detailed questions concerning the influence of coded moderator variables. Arguably the 
greatest contribution of meta-analysis is the introduction of the concepts of precision, systematicity and 
replicability to the conduct of research reviews (Abrami, Cohen, & d’ Apollonia, 1988; Bernard & Naidu, 1990). 

Method 



Definition of DE 

Our definition of DE derives from Keegan’s ( 1996) basic set of characteristics that distinguish DE 
from other forms of instruction. This definition allowed for the inclusion of both synchronous and asynchronous 
patterns of DE. 
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• The semi -permanent separation (place and/or time) of learner and instructor during planned learning events. 

• The presence of planning and preparation of the learning materials, student support services, and the final 

recognition of course completion by an educational organization. 

• The provision of two-way media to facilitate dialogue and interaction between students and the instructor 

and among students. 

Inclusion/Exclusion Criteria 



To be included in this meta-analysis, each study had to meet the following basic inclusion/exclusion 

criteria: 

1. Involve an empirical comparison of DE as defined in this meta-analysis. Studies comparing DE with 
national standards or norms, rather than a control condition, were excluded. 

2. Involve “distance from instructor” as a primary condition of the DE condition. DE with some face-to-face 
meetings (fewer than 50%) was included. 

3. Report measured outcomes for both experimental and control groups. 

4. Report sufficient statistical data. Studies with insufficient data for effect size calculations (e.g., with means 
but no standard deviations or no inferential statistics or no sample size) were excluded. 

5. Be publicly available or archived. 

6. Include at least one achievement, attitude or retention outcome measure. 

7. Include an identifiable level of learner. All levels of learners from kindergarten to adults, whether 
informally schooled or professionally trained, were admissible. 

8. Be published or presented no earlier than 1985 and no later than December of 2002. 

9. Include outcome measures that were the same or comparable. If the study explicitly stated that different 
exams were used for the experimental and control groups, the study was excluded. 

Data Sources and Search Strategies 

The studies used in this meta-analysis were located through a comprehensive search of publicly 
available literature from 1985 through December of 2002. Electronic searches were performed on the following 
databases: ABI/Inform , Compendex, Cambridge Scientific Abstracts, Canadian Research Index, Communication 
Abstracts, Digitcd Dissertations on ProQuest, Dissertation Abstracts, Education Abstracts, ERIC, Psyclnfo, and 
Social SciSearch. Web searches were performed using Google, AlltheWeb, and Teoma search engines. A 
manual search was performed in ComAbstracts, Educational Technology Abstracts; in several distance learning 
journals, including The American Journal of Distance Education, Distance Education, Journal of Distance 
Education, Open Learning, and Journal of Telemedicine and Telecare; and in several conference proceedings, 
including AECT, AACE, AERA, CADE, EdMedia, E-Learn, SITE, and WebNet. In addition, the reference lists of 
several earlier reviews were examined for possible inclusions. Although search strategies varied depending on 
the tool used, generally, search terms included “distance education,” “distance learning,” “open learning” or 
“virtual university,” AND (“traditional,” “lecture,” “face-to-face” or “comparison”). 

Outcomes of the Searches 

In total, more than 5,000 research abstracts concerning DE and traditional classroom-based instruction 
were examined and 862 full-text potential includes retrieved. Each of the studies retrieved was read by two 
researchers for possible inclusion using the inclusion/exclusion criteria. The initial inter-rater agreement as to 
inclusion was 89%. Any study that was considered for exclusion by one researcher was crosschecked by 
another researcher. Two hundred and thirty-two (232) studies met all inclusion criteria and were included in this 
meta-analysis; 630 were excluded. 



Extraction of Effect Sizes 

Effect sizes were extracted from each study. Where possible, descriptive data (i.e., means and standard 
deviations) were used to calculate Cohen’s d (see Equation 1). 



d = 

i 



Y -Y 

Experimental Control 



$ Pooled 



(i) 
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Where these data were not available, effect sizes were calculated or estimated from statistical test data 
(e.g., /-ratios, probabilities) based on equations provided by Glass, McGaw, and Smith (1981) and Hedges, 
Shymansky, and Woodworth (1989). Cohen’s d was converted to Hedges’ g (Hedges & Olkin, 1985) to correct 
for the influence of sample size. Hedges’ g+ (i.e., average effect size) was then calculated for each relevant 
subgroup and for each of the three measures along with homogeneity of effect size statistics. 

Results 

Meta-analysis is a descriptive review technique and as such provides a characterization of a large body 
of quantitative evidence. While effect sizes are tested for significance (g+ = 0), this does not involve the same 
sort of inferential testing that is often done in original research. One way of thinking of a comprehensive meta- 
analysis is as an approximation of differences in the population that far surpasses the evidence provided by any 
single study. 

Statistical Analysis 

In total, 232 studies yielding 688 independent effect sizes (i.e., outcomes) were analyzed. This was 
based on totals of 57,019 students ( k = 321) with achievement outcomes, 35,365 students ( k = 262) with attitude 
outcomes, and ( N = 57,916,029) students ( k = 105) with retention outcomes. The N reported here for retention 
was reduced proportionally to 3,744,869 to avoid overestimation based on a longitudinal California study of 
retention in junior colleges. The number of outcomes was further reduced when they were classified as either 
synchronous or asynchronous patterns. 

The statistical findings that are relevant to the question being examined in this paper are shown in Table 
1. In this table, zero defines no mean difference between the DE condition and the classroom condition. Positive 
mean effect sizes favor DE over the classroom. Negative effect sizes indicate the reverse. Bear in mind as you 
examine these mean effect sizes that wide variability surrounds each of them (i.e., homogeneity of effect size 
was violated). Hedges and Olkin (1985) warn against a strong interpretations of mean effect sizes when 
assumptions of homogeneity of effect size are violated. To dramatize this point, we found effect sizes for 
synchronous achievement outcomes to vary from -1.14 to 0.97. For asynchronous outcomes, the minimum g was 
-1.31 and the maximum g was 1.31. That says that some applications of DE, both synchronous and 
asynchronous, far outperformed their classroom comparison group and some far underperformed it. However, 
the pattern of results is interesting and bears consideration. 

First, all but two average effects sizes are significantly greater than zero. As with all statistical tests, 
higher degrees of freedom provide more power and a more sensitive test, so that at least one test (synchronous, 
retention outcomes) may represent an artifact of this. 

There is evidence for synchronous DE, based on an analysis of achievement outcomes, that students in 
the DE condition are not performing as well as their classroom companions, on average. This effect in favor of 
the classroom condition appears to be even more dramatic for attitude outcomes. By contrast, DE student 
experiencing asynchronous DE outperformed their classroom equivalents on achievement measures and 
performed equally well in terms of attitude outcomes (although the effect size is negative). 



Table 1. Weighted Mean Effect Sizes for Achievement, Attitude and Retention Outcomes (Synchronous and 
Asynchronous) 





Achievement 


Attitude 


Retention 


Categories of DE 














8+ 


SE 


g+ 


SE 


S+ 


SE 




Synchronous DE 


-0.102* 

(*=92) 


0.024 


-0.185* 
(k = 83) 


0.022 


0.005 
(* = 17) 


0.034 


Asynchronous DE 


0.053* 
(Ic = 17 4) 


0.012 


-0.003 

(*=71) 


0.019 


-0.093* 
(k = 53) 


0.021 



*p < .05 Note: All mean effect sizes are heterogeneous. 
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The situation for retention outcomes is essentially the reverse of the above. The retention rate (i.e., 
opposite of dropout) for synchronous comparisons was zero, while significantly more students dropped out of 
asynchronous DE than their classroom equivalents. This problem with DE has existed from the early days of 
correspondence education (e.g., Bernard & Amundsen, 1988). 

This analysis does not suggest that DE and classroom conditions in synchronous DE cannot be 
equivalent, just that when they have been comp ared in a large number of studies, they are not. Beyond the 
intuitive notion that remote-site students, being on the receiving end of a technologically mediated classroom 
(virtual classroom), may not feel as included, may not get to participate as much, find it harder to concentrate, 
and/or may not get the timely feedback that they need, we have little evidence to explain this apparent disparity. 
A scarcity of information in the literature, especially of the conditions in the classroom, made it difficult for us 
to find sets of robust predictors that would have helped us develop a more complete picture or empirical model 
of the instructional characteristics that make a difference. See Bernard et al., (in press) and for the complete 
analysis and Bernard, Abrami, Lou, and Borokhovski (in press) for an extended discussion of the 
methodological state of DE research. 

Study Feature Predictors of Outcomes 

In an attempt to further explain the results just presented, we used weighted multiple regression 

1/^2 

(WMR, where the weighting factor is the inverse of the population variance, 1/0 , approximated in an 

equation from Hedges & Olkin, 1985, p. 174) using individually coded study features as predictors andg as the 
outcome variable. The study features were categorized as: 1) 13 methodological predictors; 2) 9 pedagogical 
predictors; and 3) 9 media use predictors, and were entered into blocks in WMR. Methodology was entered 
first, followed by pedagogy. In a second analysis, methodology again was entered first, followed by media use. 
In this way we were able to assess the effects of pedagogy and media use, independently, after variation due to 
methodological differences was removed. We did this for synchronous and asynchronous DE outcomes 
separately. In presenting these findings, we must warn against the over-interpretation of individual study 
features. While all of the ones that are described here were significant, a certain degree of collinearity among 
them makes drawing strong conclusions problematic. 

Synchronous DE. The question that we will be attempting to answer here is: If synchronous DE and its 
classroom counterparts are not equivalent, how can we make them more so? Since from our overall analysis, we 
have established that in synchronous DE, the classroom condition had better achievement and better attitudes 
towards instruction than the DE condition, what can be added to or changed about the DE condition to 
ameliorate the situation? In looking at Table 2, which displays the significant predictors of achievement and 
attitudes (an analysis of retention outcomes produced no significant predictors), we see study features that favor 
the classroom on the left and those that favor DE on the right (this was determined by the sign of the regression 
weight associated with each predictor). 

The most striking feature contained in the set of predictors for synchronous DE is the large role that 
personal contact with the instructor and other students, either face-to-face or mediated by technology, plays in 
achievement and attitude outcomes. This agrees with much of the previous literature of DE, which cites feelings 
of isolation as a significant contributor to problems such as dropout. Two other predictors were significant, 

“Use of one-way TV-video” and “Use of systematic instructional design.” In this case, TV -video does not refer 
to the means of communication (i.e., videoconferencing). It is, instead, the use of televised materials and 
content delivered at a distance. It is conceivable that such materials offer a welcome alternative to lecture and 
other forms of teacher-dominated instruction. It is not surprising to find that explicit prior planning improves 
the quality of DE. Had there been more information about classroom conditions, we might have found the same 
thing for face-to-face instruction. 

Table 2. Summary of Study Features that Significantly Predict Outcomes in Synchronous DE 

Synchronous DE 

Favor Classroom Instruction (-) Favor Distance Education (+) 



Achievement 



Achievement 



• Face-to-face meetings with the • Face-to-face contact with other students 

instructor • u se of one-way TV-video 

• Use of the telephone to contact 
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instructor 



Attitudes 


Attitudes 


• Opportunity for face-to-face contact with 


• Use of systematic ID 


other students 


• Opportunity for mediated communication 


• Use of one-way TV-video 


with the instructor 




• Instructor/student contact encouraged 

• Use of the telephone to contact instructor 



Asynchronous DE. Table 3 is structured in the same way as the previous table. Here again, 
communication and the use of mediated content delivery play important roles. However, in contrast to 
synchronous DE, an explicit learning strategy, “Problem-based learning” emerged as an important predictor of 
both achievement and attitude outcomes. This is particularly interesting when combined with the appearance of 
computer-mediated communication. It is possible that this pairing suggests a positive effect for computer-based 
collaborative learning. 

Table 3 . Summary of Study Features that Significantly Predict Outcomes in Asynchronous DE 

Asynchronous DE 

Favor Classroom Instruction (-) Favor Distance Education (+) 



Achievement 


Achievement 


No significant predictors 


• Use of problem-based learning strategies 

• Opportunity for mediated 




communication with the instructor 




• Advance information given to students 

• Use of one-way TV-video 



Attitudes 


Attitudes 


• Use of the Web 


• Use of problem-based learning strategies 

• Use of computer-mediated 
communication 

• Use of computer-based instruction 



Discussion 

One of the things that meta-analysts quickly come to realize is that they are “prisoners” of the data and 
the previous efforts of primary researchers. Findings are based, by necessity, on what can be gleaned fromthe 
literature; nothing new can be added. In this study, lack of study information, particularly regarding to the 
unreported characteristics of the classroom condition, frustrated our efforts to delve more deeply into the nature 
of instructional study features and their relationship to the overall findings. Why do effect sizes vary so widely, 
even after they are categorized as synchronous or asynchronous? What should be included in synchronous DE 
to make it as effective as (equivalent to) classroom instruction, or at least the best it can be? What practices 
should be avoided? What are the ranges of instructional practices and learning strategies that best support 
achievement and satisfaction in asynchronous DE? Answers to these kinds of questions did not emerge as 
clearly as they could have if the literature were more complete. 

Nevertheless, some interesting findings did emerge concerning the nature of synchronous and 
asynchronous DE. We found that synchronous DE, in particular, produced average effect sizes that favored the 
classroom condition for both achievement and attitude outcomes. We also found wide variation around these 
means. This suggests that synchronized DE classrooms can produce outcomes that are at least equivalent to 
“live” classrooms but that this doesn’t always happen (in fact, in a majority of cases it doesn’t). The reasons for 
this may be manifold, ranging from the application of teacher-centered instructional techniques, which might 
not be as engaging in a mediated form, to failures associated with the technologies involved (either through 
poor application or failures of the technology itself). It is very likely that a new set of instructional skills, 
beyond those applied in the classroom, is required for instructors to meet the challenges of synchronous DE. 
While the tested principles of pedagogy (e.g., motivation, engagement, interactivity, evaluation, feedback) may 
generally apply, their application in synchronous mediated DE environments may require experience and 
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possibly even special training. It is likely that many of the students involved in these studies had little if any 
prior experience with DE, and consequently did not know what to expect from DE or how to engage effectively 
in this new form of educational experience. If that is the case, this situation may change radically as more 
learning opportunities are offered and more students partake of them. 

One of the prescriptions that did emerge regarding synchronous DE, in particular, is that it should 
involve more direct personalized contact between students and the instructor and among students. This is 
tantamount to saying: “make synchronous DE more like face-to-face instruction,” and it is arguable that this is 
not a bad idea if in fact synchronous DE is to be regarded as a special case of face-to-face instruction. If this is 
not feasible, as it often is not in DE, serious efforts should be made to compensate for this deficit through 
various mediated options (Simonson, Schlosser & Hanson make this very point). 

We also found some interesting relationships regarding asynchronous DE. In another paper (Bernard et 
ah, in press) we provide evidence of the precedence that pedagogy takes over media in DE, especially in 
asynchronous settings, a point that Richard Clark ( 1983, 1994) has made for years regarding all forms of 
technology applied to instruction. While it is axiomatic that DE requires technology, it must function in the 
service of something — content delivery (e.g., TV/video, Web resources), communication between student and 
instructor (e.g., clarification and feedback), and/or communication among students (e.g., mediated 
communication) — that leads to learning success. We have found direct evidence here of all of these elements. 
We have found indirect evidence that collaborative learning (i.e., the combination of mediated communication 
and problem-based learning) is present. Direct evidence of the effectiveness of collaborative learning strategies 
may emerge as more studies of this relatively recent approach to learning are conducted. It is also possible that 
advances in multi-media, CBI, simulation media etc. may herald the new era of interactiveness posited by 
Ullmer (1984) and Cobb (1997). 

This empirical assessment of the comparative state of affairs in studies of synchronous and 
asynchronous DE begs the question of whether a new theory — “equivalency theory” or any other, for that 
matter — is required to understand and to make progress in the development of DE. Meta-analysis cannot answer 
such questions, even when the question involves a testable hypothesis. This meta-analysis has demonstrated, 
however, that we have a long way to go before we can say with certainty that either form of DE will reliably 
offer educational opportunities that equal or exceed that which is currently called “traditional classroom 
instruction.” 
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